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Summary 

This report describes an evaluation of Computer Literacy, which is an undergraduate general studies 
course, offered by College of Education at a large southwestern university. The purpose of this course is to 
provide knowledge about computer and computing, and application skills in using Microsoft Office software. 
The course is coordinated by a professor and is currently delivered in multiple sections by teaching assistants. 

The evaluation employed a questionnaire survey and an interview as the primary data collection 
methods. In total, 329 students and all 11 instructors responded to the survey. The interview data were collected 
from five student focus groups, five instructors, and the course coordinator. In addition, mean scores of quizzes, 
the midterm exam and the final were obtained from all class sections. 

The survey and interview results suggested that the topics taught were useful to students, especially the 
hands-on skills. Among the different teaching strategies used, hands-on projects and in-class activities were 
found to be more helpful to students, whereas reading from textbook and online discussion forum were not 
helpful. The results suggest that ‘learning by doing’ is good for learning skills, whereas reading and group work 
are not helpful unless they can make the class more interesting. 

Overall, the evaluation revealed that the Computer Literacy course is successful in providing basic 
understanding and application of computer. Recommendations on alternative teaching strategies are made to 
improve learning effectiveness. 



Introduction 

Computer literacy is expected for both academic and career achievement (Davis, 1999). Instructors at 
school expect their students to have some degree of computer literacy when they enter college (Hirschbuhl & 
Faseyitan, 1994) and when they graduate (Furst-Bowe, Boger & Franklin, 1995). In the job market, corporate 
recruiters have reported that core computer competency skills are very important to the employability of a 
recent college graduate (Davis, 1997). Ndahi and Gupta (2000) explained that this is because computer literate 
employees have better prospective in workplace-specific training and more likely to be successful in their field. 
As a result, employers seek computer literacy in almost everyone their hire (Ndahi & Gupta, 2000). Thus, a job 
applicant who lacks computer skills is seriously handicapped, both in obtaining a job and qualifying for 
promotion (Martin, Carrier & Hill, 1997). 

To meet the demand of computer-literate graduates, it is important to determine what constitutes 
desired computer competency and how they should be taught. Tucker and Garnick ( 1991) argued that a 
computer literacy course should be characterized to emphasize using computers as tools alongside the uses of 
computers in society. Ndahi and Gupta (2000) conducted a survey on computer literacy in workplace training, 
and the results suggested that word processing was the most required skill. The report also showed a strong 
interest in learning file management and making a PowerPoint presentation. The demand for knowing database 
software was also increasing. These results were consistent with Furst-Bowe, Boger and Franklin’s study 
(1995), which indicated that skills in word processing, spreadsheets, database management, graphics, and 
information retrieval are required for many jobs. In terms of teaching strategies, previous studies found that 
students learn more when they are allowed to work in a lab together, or when informal peer tutoring and support 
are available (Davis, 1999). Learners also prefer to customize learning at their own pace (Smith & Tarkow, 
1998). 

Course description 

The evaluated Computer Literacy course is a 3 -credit general studies course required for undergraduate 
students. In the 14 three-hour weekly meetings with instructors, students learn not only fundamental concepts 
and knowledge about computers, such as how data are input, processed, stored and output (IPSO), but also how 
to actually use computer applications for productivity, such as using Microsoft Word, Excel and PowerPoint. 
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The administering of the course incorporates Blackboard, a course management system widely used on campus. 
All course information, including presentation notes, hand-outs, assignments, and quizzes/exams can be 
accessed from Blackboard. 

In spring 2003, there were 444 students enrolled in 21 sections of the Computer Literacy course taught 
by 1 1 teaching assistants. The class size of each section ranged from 18 to 24 people. One professor coordinated 
all these sections. A formal evaluation was conducted to improve learning effectiveness. 

Elements evaluated include the contents taught, the skills learned, and the teaching strategies used in 
the course. Student and instructor perceptions of the course were also noted. 

Method 

Participants 

The participants in the evaluation were the students, instructors, and coordinator of the Computer 
Literacy course. An online survey was administered to the 444 undergraduate college students enrolled in the 
course. Twenty-five students were interviewed in focus groups. They were volunteers from five different 
sections with five students in each group. The 1 1 instructors who taught this course were surveyed and five of 
them were interviewed. The course coordinator was also interviewed. 

Data sources 

A variety of evaluation instruments were employed to assess the course in a comprehensive manner. 
Data were collected with regards to the contents covered, skills learned, teaching strategies used, and student 
and instructor attitude towards the course. The instruments include: 1) Student survey, 2) Instructor survey, 3) 
Student interview of the focus groups, 4) Interview of instructors, 5) Interview of the course coordinator, and 6) 
Test scores of all 21 sections. 

Student survey . The 26-item student survey was designed to determine the general perception of the 
Computer Literacy course in terms of the contents covered and teaching strategies used. The survey was 
distributed through Blackboard to all students enrolled in the course. Responses to the questions were 
anonymous and were scored on a four-point Likert type scale from “most agree with the description (3)” to 
“least agree with the description (0)”. The respondents were also given the opportunity to provide additional 
comments. 

Instructor survey. The instructors were administered a paper-based survey that was similar to the 
student survey. Their attitudes towards the content and the teaching strategies were rated on the same four-point 
Likert type scale. 

Interview protocol. Interviews with the instructors, focus group students, and the course coordinator 
were designed to collect detailed information on topics covered by the course, teaching strategies used, and 
student learning gain at the end of the semester. The questions were directly aligned to the questions on the 
survey. All interviewees were given the chance to address any issue relevant to the course. 

Test scores. Student performance on the online quizzes, online midterm exam, and hands-on final were 
analyzed to determine the usefulness of content and helpfulness of strategies. The online tests were multiple - 
choice questions that assess student knowledge about computers and skills of using a computer. The hands-on 
final exam tested students how to use Microsoft Word, Excel, and PowerPoint. Students needed to complete 
editing, analysis, and design activities as requested. The four 10-point quizzes covered the primary topics of the 
course, namely: Quiz 1 - File Management, Word, Internet and World Wide Web, Quiz 2 - IPSO, Quiz 3 - 
Excel and Data Analysis, and Quiz 4 - PowerPoint. Topics covered during the first half of the semester (Quiz 1 
and 2) were tested in the online midterm exam. The maximum score of the online midterm and the Hands-on 
final was 30 points. 



Results 

Survey responses. Based on a four-point Likert -type scale scoring from 3 (most agree with the 
description) to 0 (least agree with the description), the survey responses of 329 students and 1 1 instructors on 
the usefulness of the contents are shown in Table 1, and the helpfulness of teaching strategies are shown in 
Table 2. 

In terms of the usefulness of the contents taught, student and instructor survey responses revealed the 
topics covered in this course are generally considered very useful or useful (student rating M = 2.08, instmctor 
rating M = 2.44). Among the seven items that were rated, Microsoft Word , PowerPoint, Internet and World 
Wide Web were the top three highest ratings across students and instructors. Knowledge on IPSO was rated the 
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least useful (student rating M = 1.53, instructor rating M = 1.64). It is worth noting that there was a big 
diffeience in the perceived usefulness on File Management between students (M = 1.74) and instructors ( M = 
2.45). 

In terms of teaching strategies, both students and instructors ranked that Hands-on Projects , In-class 
Activities to Develop Practical Skills, and Handouts for Activities and Projects were the three most helpful 
strategies. Students and instructors agreed that the two least helpful strategies are Readings by students (student 
rating M = .73, instructor rating M = 1.09) and Online Discussion Forums (student M = 1.22, instructor rating M 
= 0.82). 

The responses to the open-ended question at the end of the survey are summarized in Table 3. Twenty- 
nine people said they were satisfied with the course and no change needed to be made. There were also 
suggestions in terms of contents taught and teaching strategies used. Many students preferred to have more 
hands-on activities in class (N = 36) and get more individual attention from the instructor (N = 21). Students 
also commented that lectures in the form of PowerPoint presentations were too long (TV = 19). The group Online 
Discussion Forum was considered not helpful (N = 12) due to the difficulty of cooperation among group 
members. Students also considered reading textbook as not helpful ( N = 12), and the quiz questions did not test 
the skills learned but how much details remembered (TV =6). 

Interviews. The results of the interviews of instructors and student focus groups were consistent with 
those of the survey on the topics covered in the course. In addition, both students and instructors emphasized 
that more practices were needed for File Management. Specific step-by-step instructions were needed for 
complicated subjects such as Excel and Create a Personal Website. 

The interview with the course coordinator revealed that this course is meant to provide fundamental 
knowledge and skills that students can start with. Software application skills (Microsoft Word, Excel, 
PowerPoint, and Create a Personal Website ) were long-term gains for the students. It was difficult to maintain 
consistency of teaching across 21 sections and 1 1 instructors. PowerPoint presentations were pre -designed and 
distributed to instructors to maintain the consistency. 

Text scores The mean scores and percentages of achievement of all tests across the 21 sections 
are summarized in Table 4. Among the online tests, students performed the best in Quiz 2 on IPSO (M = 8.14) 
and Quiz 3 on Excel (M = 8.09). The highest achievement was their hands-on final exam (M = 26.17), which is 
a 87. 23% of the maximum score. 

Discussion 

What to teach? 

According to the survey results, the contents taught in the course achieved a rating of 2.26 across 
students and instructors, indicating that the topics covered by the Computer Literacy course are useful or very 
useful. However, there appears to be a big difference in learner interest of learning knowledge and learning 
application skills. It seems that application skills (M = 2.44), including Word, Excel, PowerPoint, and Create a 
Personal Website, were rated more useful than concept knowledge (M = 2.04) that consisted of File 
Management, Internet and World Wide Web, IPSO. Does this mean concept knowledge is less important than 
application skills? 

Interestingly, the concept of File Management was rated 2.45 by the instructors, 0.7 1 point higher than 
student rating (M = 1.74). In addition, students in the interview suggested more practice on File Management. 
This might be because many students were not able to realize that mastering File Management is essential to 
working with their documents correctly and efficiently. Poor file management may not be a problem until late in 
the semester when student have to hyperlink all their assignments to their personal websites. If this is the case, 
how can the concept knowledge be taught in a more interesting way? Can the knowledge be integrated with the 
teaching of application skills? 

How to teach? 

The survey results showed that all of the three hands-on strategies ( Hands-on projects. In-class 
activities, Handouts for activities and projects ) were helpful. In addition, the overall rating of hands-on 
strategies (M = 2.53) is much higher than the overall rating of other strategies (M = 1 .48). This result is 
consistent with student and instructor interview. It indicates thatskills are learned through hands-on practice 
with instructor feedback. In order to give students more attention in class, the class size should be small enough 
for the instructor to provide individual assistance. For a class that has 18 to 24 students, it might be better to 
downsize the class if there is only one instructor available. Alternatively, there can be a second instructor added 
to assist hands-on practices. 

Both the survey results and interview revealed that the current Online Discussion Forum is not a 
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preferred teaching strategy. Some student reported that the group work did not work as it was supposed to. This 
might be because students have different schedules and it is hard to work as a group after class. To improve 
collaboration among students, it is advisable to provide more opportunities in class to work together. 
Furthermore, each student should have a clear idea of his or her responsibility and how their job will be rated. 
Instructor supervision over the discussion forum needs to be enhanced. Guidance and feedback provided by the 
instructor will also motivate students to do a better job. 

An interesting phenomenon is that the type of assessment may influence student performance. As 
shown in Table 4, student’s achievement in the online tests was lower than the final hands-on exam. Students 
reported that the online quizzes were “too detailed” and “did not test the skills learned” but “how much is 
memorized”. The majority of online test questions were multiple -choice format, which was used to assess 
concept knowledge and practical application skills. Since the skills would be tested without actually using the 
application software, the multiple -choice format seems to have changed the nature of application skills from 
“hands-on” to memory -based. Consequently, student performance was hampered by this type of assessment. 
This may also explain why Quiz 2 had the highest mean scores among all the online tests since the subject of 
Input, Processing, Storage, and Output was mostly factual knowledge as opposed to application skills . Along 
the same line, the reason why Quiz 3 on Excel and Data Analysis had achieved a high average might be because 
the questions were not just theoretical knowledge but it also tested some hands on excel skills , and students had 
to do calculations using Excel worksheet. In this case, students could actually work in the software instead of 
recalling where to access a command and under which menu. It appears that students perform better when 
hands-on skills are assessed in a hands-on way. 

Recommendations 

The Computer Literacy course is considered as a good general studies course that provides students with basic 
understanding of computer and application skills to use a computer. Based on the evaluation results, the 
following recommendations are made: 

1. Reinforce the understanding of concept knowledge in teaching application skills. 

2. More in-class, hands-on activities should be emphasized in teaching facilitated by appropriate student- 
instructor ratio. 

3. Online discussion forums can be used for sharing experience of using application software. 

4. Use hands-on tests to assess application skills. 
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Table 1 Student and Instructor Survey Mean Scores on the Usefulness of Content 



Application Skills 


Students 


Instructors 


Total 


Word 


2.53 


2.73 


2.63 


Excel 


2.13 


2.55 


2.34 


PowerPoint 


2.35 


2.82 


2.59 


Create a Personal Web Site 


2.01 


2.36 


2.19 


Subtotal 


2.26 


2.62 


2.44 


Concept Knowledge 


File Management 


1.74 


2.45 


2.10 


Internet and World Wide Web 


2.30 


2.55 


2.43 


Input, Processing, Storage, Output 


1.53 


1.64 


1.59 


Subtotal 


1.86 


2.21 


2.04 


Total 


2.08 


2.44 


2.26 



Table 2 Student and Instructor Ratings of the Helpfulness of Teaching Strategies in the Course 



Hands-on Strategies 


Students 


Instructors 


Total 


Hands-on projects 


2.52 


2.91 


2.72 


In-class activities 


2.28 


2.64 


2.46 


Handouts for activities and projects 


2.27 


2.55 


2.41 


Subtotal 


2.36 


2.70 


2.53 


Other Strategies 


PowerPoint presentations to deliver lecture 


2.12 


1.82 


1.97 


Online multiple -choice tests 


1.84 


2.09 


1.97 


External website links 


1.69 


1.64 


1.67 


Co-operative group work 


1.34 


1.27 


1.31 


Online discussion forums 


1.22 


0.82 


1.02 


Reading textbooks 


0.73 


1.09 


0.91 


Subtotal 


1.49 


1.46 


1.48 


Total 


1.78 


1.87 


1.83 



288 












Table 3 A Summary of the Answers to the Opened-ended Question in Student Survey 



Student Comments 


Frequency 


1 . 


Satisfied with the content taught; the course was well organized; no changes need 
to be made. 


29 


2. 


To include more hands-on practice/examples or step-by-step instructions in class, 
especially when teaching complicated subjects. 


36 


3. 


Need more than one instructor for a class size of 18-24 people; instructors should 
give more individual attention to students. 


21 


4. 


The lectures and presentations were long and boring; demonstrations are more 
interesting. 


19 


5. 


The group Online Discussion Forum should be removed or changed; the group 
activity designed did not work. 


12 


6. 


Reading from the textbook had little value. 


12 


7. 


Split the weekly three-hour class into two shorter meetings. 


9 


8. 


The quizzes were too detailed; the quizzes did not reflect skills learned but tested 
memorization. 


6 


9. 


To include more collaborative activities to make the class more interesting. 


6 



Table 4 Mean Scores of Quizzes and Exams across All Sections 



Quiz 1 


Quiz 2 


Midterm 


Quiz 3 


Quiz 4 


Hands-on Final 


Word, File 


Input, 


Including Quiz 


Excel and 


PowerPoint 


Word, Excel, 


Management, 


Processing, 


1 and Quiz 2 


Data Analysis 




PowerPoint 


Internet and 


Storage, 










WWW 


Output 










6.85 


8.14 


22.83 


8.10 


7.00 


25.57 


69% 


81% 


76% 


81% 


70% 


85.23% 
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